Existing video compression systems can compress a stream of video data so it takes less bandwidth to send over a communication channel. Such systems take advantage of redundancies expected to occur in the video they are aiming to compress. For example, JPEG and MPEG take advantage of frequent similarities in the colors of adjacent pixels in photographic images. In addition, MPEG takes advantage of the fact that motion pictures often have many pixels that stay the same color during many frames of video or only shift their positions on the screen as the camera moves.
Video can be further compressed depending on how much degradation in video quality (or “video loss”) is acceptable to the person (or “user”) viewing the video, but the acceptability of different types of video loss is highly dependent on the user's activity (or “application”). The four types of video loss are; (1) resolution loss (appears blurred), (2) color depth loss (has fewer shades of colors), (3) frame rate loss (stalling or jerkiness of a motion picture) and (4) time loss or “video delay” (time delay from video capture to its availability for viewing).
To achieve higher compression ratios, different compression systems take advantage of the types of video loss that are the most acceptable to the users they aim to satisfy. For example, with MPEG, fast action scenes that would generate too much data for the communication channel are sent with resolution loss because movie viewers accept resolution loss better than they accept frame rate loss or color depth loss.
Video delay is not a problem in some applications but it is a serious problem in other applications. Different compression systems impose different amounts of delay as they compress the video. Systems that impose more delay achieve higher compression ratios because all the video frames captured, held and examined during the delay provide a better opportunity to decide how to compress them. One example might be: “is the camera moving or is just one object in the scene moving”.
Video delay is not a problem with “one-way” user activities, such as watching movies; therefore, the compression systems used for these applications (such as MPEG) impose a long delay (many seconds or more) before compressing the video and beginning to send it over the communication channel. In fact, when the communication channel is a network with indeterminate bandwidth availability (such as the Internet), the video received from the network is often buffered and delayed for many more seconds before it is displayed (to eliminate the stalling caused by network congestion). Although time delay is not a problem with one-way user activities such as watching movies, it is a serious problem for real time “interactive” users, such as users with a mouse, controlling a cursor that is a part of the compressed video image.
One such example of real time interactive users relates to the remoting of a computer KVM console (Keyboard, Video display and Mouse) over a communication channel. In these “remote console” applications, keyboard and mouse data are sent from the remote console over the communication channel and “switched” to one of a number of “target” server computers, just as if the keyboard and mouse were directly connected to that target server. The corresponding video is sent from the target server to the remote console just as if the target server was directly connected to the remote console's video display. Examples of KVM systems are described in commonly-owned U.S. Pat. Nos. 5,721,842 to Beasley et al and 5,732,212 to Perholtz et al, each of which is incorporated herein by reference.
The communication channel for some KVM systems provides enough bandwidth to transport the uncompressed video because they use dedicated local cables and a dedicated circuit switch. KVM systems adapted to operate over a network via, for example, Internet protocol (referred to herein for brevity as “KVM/IP” systems) provide limited and indeterminate bandwidth availability compared to a dedicated local cable-based KVM system. Sending keyboard and mouse information from the remote console to the selected target server in a timely fashion is one concern with KVM/IP systems. A greater concern is sending the relatively high volume of video data back to the remote console in a timely fashion. Since today's typical computers output video continuously at over 2 gigabits per second and remote Internet connections (such as DSL) typically operate at less than 1 megabit per second, video compression ratios averaging well over 2000-to-1 are required. Remote Internet connections using dial modems at 50 kilobits per second require even higher average compression ratios.
As a remote console user moves their mouse or types on their keyboard to input new information to the server, those actions must be communicated to the server and acted upon by the server to create new video images, which are sent back to the remote console user's screen. Delays in sending the video back to the remote console user are annoying because they create a temporal lag between the entry of the keyboard or mouse information by the user and the video response perceived by the user on their screen. Delays following keyboard activity are less annoying than delays following mouse movements, thus the term “mouse-cursor response” is used to describe this problem.
This problem of remote console applications (described above) is not applicable to some types of typical web browser applications. With web browser applications, the video cursor image is created locally on the user's computer, so mouse-cursor response is always very good even if the network is slow at responding with server-generated video images. With remote console applications, network delays affect the mouse-cursor response because the cursor is represented as an integral part of the video image coming from the server and sent to the remote console over the network.
In remote console applications, user acceptability for the four types of video loss is the complete opposite from other video applications. As described above, minimum video time delay is a factor in remote console applications, but video delay is a less important type of video loss in other applications. The importance of resolution loss in remote console applications is also the opposite of other applications because the computer screens sent to remote consoles are typically made up of a significant amount of relatively small font alphanumeric text, many small icons and many high contrast sharp edges. Compression systems such as JPEG or MPEG, that impose resolution loss may be satisfactory for many other applications, but they are not satisfactory for reading small font alphanumeric text and images with high contrast sharp edges. The opposite order of user acceptability also applies to color depth loss and frame rate loss. These two types of video loss are the most acceptable by users in remote console applications and the least acceptable in other video applications.
Although existing video compression systems are widely used and well suited for a wide variety of applications, a video compression system optimized for the best possible interactive computer user experience is needed.